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We present a data-basedperspective concerning the recent Macleans magazine rankings of Canadian 
universities, including cluster and other analyses of the 2007and2008 data. Canadian universities 
empirically resemble and relate to each other in a manner different from their formal classification 
and final rank ordering in the Macleans system. Several pitfalls in ranking procedures, related to 
invalid and unreliable relationships among indices underlying the final ranks, are outlined, along 
with relevant findings from previous studies. In their present format, although they have become in¬ 
creasingly publicized and promoted, data based on the Macleans system are of limited practical use 
to students. Perhaps more important, ranking exercises have unintended though potentially serious 
consequences in terms of the intellectual and personal well-being of students. 


Introduction 

I n November 2007 and December 2008, Macleans 
published its 18th and 19th annual rankings of 
Canadian universities. Indeed, the ranking of univer¬ 
sities has become a popular exercise with which to 
assess and promote higher education in North Amer¬ 
ica (e.g., Bruneau & Savage, 2001; Cramer & Page, 
2007; Page & Cramer, 2002, 2003; Page, Cramer, & 
Page, 2008; Provan & Abercromby, 2000). The rank¬ 


ing approach is similar to that used by publications 
such as Consumer Reports, in which goods or services 
are assigned scores, and then assigned relative rank 
standings. 

In this paper, we summarize our examination 
of the 2007 and 2008 ranking data, and present im¬ 
plications of ranking exercises for the issue of student 
welfare. 
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Ranking Procedures 

For this exercise, Macleans classifies universities (N = 
47) into three types: Medical/Doctoral universities 
(N = 15), which contain medical schools and have 
large graduate departments; Comprehensive univer¬ 
sities (N = 11), which have no medical schools but 
may have graduate programs, and Primarily Under¬ 
graduate universities (N = 21). 

The Macleans ranking data have always re¬ 
flected six main measures: Student Body (including 
indices of student ability); Classes (including indices 
of class size and percentage of classes taught by ten¬ 
ured faculty); Faculty (indices of faculty members’ 
academic characteristics; Finance (indices of budget 
parameters and student services; Library (indices as¬ 
sessing holdings); and Reputation (indices based on 
alumni support and a reputational survey). Pooled 
over the six measures, Macleans has typically used 
a total of 24 indices for Medical/Doctoral universi¬ 
ties, 23 for Comprehensive universities, and 22 for 
Primarily Undergraduate universities. For 2007 and 
2008, however, rankings were reduced to 14, 13, and 
13 indices respectively, though were still grouped 
within the above six measures. Details about each in¬ 
dex are available in every Macleans annual university 
rankings issue. 

Macleans compiles preliminary data for each 
index separately, computes a weighted sum of points, 
then constructs an overall rank, with standings re¬ 
ported separately within each university type. In the 
2007 data, McGill ranked highest in the Medical/ 
Doctoral category. For Comprehensive schools, the 
University of Victoria ranked highest. For Under¬ 
graduate schools, Acadia and Mount Allison tied for 
highest. For 2008, McGill, Simon Fraser, and Mount 
Allison ranked highest in these categories, respective- 

J y- 

In previous studies, we have examined the 
Macleans ranking parameters and published data, 
including the extent to which the indices were in- 
tercorrelated and related to the overall final rankings 
assigned. We also have examined the degree to which 
higher-ranking universities were reliably different 
from those of lower rank. We have found consistent¬ 
ly that the main measures and component indices 


used by Macleans are internally inconsistent and not 
reliably related either to each other or to final ranks. 

Observations: 2007 and 2008 
Ranking Data 

For 2007, using Spearman rho (rank-based) correla¬ 
tions, we find that many indices are unrelated to fi¬ 
nal rankings. The proportion of indices significantly 
related (i.e., at p < .05) to final rank, collapsing over 
the three university types, was .471. 

For Medical/Doctoral universities, 8/14 in¬ 
dices (57.12 %) were related significantly to final 
rank. For Comprehensive universities, 4/13 (30.76 
%) were related significantly to final rank. For Un¬ 
dergraduate universities, 7/13 (53.84 %) were related 
significantly to final rank. For each university type, 
as in previous studies, many of the rho correlations 
were negative , that is, with higher final rankings cor¬ 
related with lower rankings on certain indices, and 
vice versa. 

Although they are conceptually similar across 
the three university types, the specific Macleans in¬ 
dices in 2007 and 2008 empirically correlate weakly 
with each other, and also elicit different patterns of 
interrelationships and interpretations, depending on 
the university category under consideration. 

For 2008, using Spearman rho (rank-based) 
correlations, we find again that many indices are un¬ 
related to final rankings. The proportion of indices 
significantly related (i.e., at p < .05) to final rank (11 
of 14), collapsing over the three university types (N 
= 47) was .712. We note that relative rankings, as 
well as correlations between rankings and reputation, 
continue to change dynamically each year. Moreover, 
due to minor variations annually in the underlying 
data over time, the top several schools in each cat¬ 
egory necessarily shift back and forth in terms of 
final rank standings, relative to each other. For Un¬ 
dergraduate schools, the rho correlation between the 
2006 and 2007 rankings was .655, indicating that 
most of the 2007 rankings were not predictable di¬ 
rectly from those of the previous year. For 2008, this 
correlation was higher, i.e., .942 (p < .0001). 

For Medical/Doctoral universities in 2008, 
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8/14 indices (57.12 %) were related significantly 
to final rank. For Comprehensive universities, 4/13 
(30.76 %) were related significantly to final rank. For 
Undergraduate universities, 5/13 (38.46 %) were re¬ 
lated significantly to final rank. For each university 
type, as in previous studies, many of the rho correla¬ 
tions again were negative, that is, with higher final 
rankings correlated with lower rankings on several 
indices. 

Higher Versus Lower-Ranking 
Universities 

We assessed to what extent lower-ranking universities 
in 2007 differed from higher-ranking ones, in terms 
of the various indices in the Macleans system. The 
top and bottom subgroups (halves) of the universi¬ 
ties, within each type, were therefore compared using 
the Wilcoxon Rank Sum (Mann-Whitney U) test. 
The Wilcoxon test examines the significance of dif¬ 
ferences in ranked data on a specified index, taken 
from two independent samples (universities). For all 
universities pooled together, 12 comparisons of these 
indices were significant. 

For Medical/Doctoral universities, the top 
and bottom groups (halves) differed significantly, at 
p < .05, on 6/14 individual indices (42.85%). For 
Comprehensive universities, the top versus bottom 
halves differed significantly on 3/13 (23.07%) of 
these indices. For Undergraduate universities, the top 
versus bottom halves differed on 3/13 (23.07%) of 
the indices. Thus, collapsing over the three university 
types, the top and bottom halves did not differ sig¬ 
nificantly in average rank on 28 of the 40 individual 
comparisons. For these comparisons, higher-ranking 
universities therefore had little or no difference from 
those of lower rank. 

We note in passing that conventional math¬ 
ematical operations, quantitative comparisons, or 
interpretations of differences, i.e., beyond observing 
the rank differences per se, are not possible with or¬ 
dinal (i.e., rank-based) data (Siegel, 1959). 

For the 2008 ranking data, the indices for the 
top and bottom subgroups (halves) of the universi¬ 
ties, within each type, were again compared. For all 


universities pooled together, 14 of these 40 compari¬ 
sons (35 %) were significant. 

For Medical/Doctoral universities, the top 
and bottom groups (halves) differed significantly, at 
p < .05, on 5 of 14 individual indices (35.71 %). For 
Comprehensive universities, the top versus bottom 
halves differed significantly on three of 13 (23.07 
%), and for Undergraduate universities, the top ver¬ 
sus bottom halves differed on six of 13 (46.15 7%) 
of the indices. Thus, collapsing over the three uni¬ 
versity types, the top and bottom halves did not dif¬ 
fer significantly in average rank on 26 of the 40 (65 
%) individual comparisons. For these comparisons, 
higher-ranking universities therefore had little or no 
difference from those of lower rank. 

Cluster Analysis of Universities 

A vertical rank ordering of schools tends to exagger¬ 
ate the apparent differences in final ranks, encourages 
over-interpretation of rank differences, and obscures 
similarities. From a different perspective, we therefore 
employed a cluster analysis (Landau & Leese, 2001) 
with which to examine patterns of interrelationship 
among the universities for the 2007 and 2008 rank¬ 
ings, across the three university types used by Ma¬ 
clean’s. Using algorithms outlined by Ward (1963), 
squared Euclidian distances, as estimates of distance 
between schools, were calculated for the 47 schools, 
based on their raw scores for the 14 (13 for Compre¬ 
hensive; 13 for Undergraduate universities) indices. 
The overall analysis for the 2007 rankings, including 
all universities, yielded five primary clusters. Unique 
clusters of schools were identified, in which the simi¬ 
larity of each member’s corresponding profile was 
maximized, and intercorrelations among members 
were high. Schools were thus highly similar within a 
cluster, but dissimilar to those outside of the cluster. 

Generally, members within each cluster 
showed considerable variation in their final rankings. 
In several cases, universities also belonged to clusters 
in a manner inconsistent with their membership in 
one of the three university categories. Thus, one may 
distinguish between formal a priori or conceptual sim¬ 
ilarity of universities within a particular type, as de- 
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fined by Macleans, and empirical similarity as defined 
by quantitative analysis of the ranking indices them¬ 
selves. In most cases, the pattern of relationships with¬ 
in and between clusters was not clearly reflective of 
rank differences between higher- and lower-standing 
universities within the three university types. In effect 
then, schools of different characteristics, programs, 
types, and rank standings may nevertheless be similar 
in terms of their scores on a particular set of indices. 

For the 2008 data, the overall analysis, in¬ 
cluding all universities, again yielded five primary 
clusters. Generally, members within each cluster 
could be grouped into similar pairs or subclusters, 
and showed considerable variation in their final rank¬ 
ings and membership with reference to the three Ma¬ 
cleans university categories. In several cases, ‘unlikely’ 
pairs of schools turned out to be empirically similar. 
In effect then, schools of different characteristics, pro¬ 
grams, missions, types, and rank standings may nev¬ 
ertheless show communality in their pattern of scores 
on a particular - arguably arbitrary - set of indices. 

Discussion and Conclusions 

Aside from the formal concerns of statistical data and 
comparisons, there remain other issues relevant to 
ranking exercises. 

First, the ranking indices and specialized aca¬ 
demic measures used by Macleans are generally in- 
congruent with the practical reasons for university 
attendance and selection typically reported by un¬ 
dergraduates themselves (e.g., Cramer & Page, 2007; 
Page, Cramer, & Page, 2008). 

Second, the annual rankings have not gen¬ 
erally reflected currently available studies of student 
satisfaction, wherein most students have indicated 
high levels of satisfaction with their own institutions, 
yet in which the highest ranking institutions, perhaps 
surprisingly, have often done relatively poorly. It is 
worth noting that, were a set of university rankings 
to be based on student satisfaction indices general¬ 
ly, it is likely that few if any significant differences 
in university-rank ordering would therefore be ob¬ 
served. 

Third, an emphasis on rank standings ob¬ 


scures the existence of individual programs and the 
unique missions of different schools, in the context 
of their particular location and type of student popu¬ 
lation. 

Fourth, among many possible side-effects of 
ranking exercises, concerns their unintended effects 
upon the quality of a university’s academic and intel¬ 
lectual spirit as they are experienced and perceived by 
students. These effects raise the possibility that the 
rankings may help to generate yet another form of 
the educational self-fulfilling prophecy at the post¬ 
secondary level (Blum, 1978; Page & Rosenthal, 
1990; Rosenthal & Jacobson, 1968), indeed one 
which will affect some students adversely, and oth¬ 
ers positively. In very practical terms, students do 
not want to believe that their schools can be easily 
and publicly rank-ordered, no more than they be¬ 
lieve that people should be ranked with indices de¬ 
fining them as haves or have-nots. Accordingly, the 
academic performance, general orientation, and per¬ 
haps overall world view, of those attending ‘lower’ 
universities continue to be threatened by exposure 
and reference to ranking exercises, and the typical 
interpretations which accompany them. Moreover, 
sizeable research literature now exists which has iden¬ 
tified less effective thinking and less efficient intellec¬ 
tual performance in individuals who feel put down 
or stigmatized in some way. A recent series of studies 
has shown, for example, that being a member of a 
negatively stereotyped racial minority can adversely 
affect one’s academic performance in situations in 
which the person believes that his or her intellectual 
ability is being assessed (Steele, 2004; Steele & Ar¬ 
onson, 1998; Kaplan & Saccuzzo, 2001). The no¬ 
tion that one’s university ranking may be used to 
calibrate levels of ‘smartness,’ (as in the Macleans 
question: “Where are the smart students?”), some¬ 
thing like finding one’s proper shoe size thus appears 
misguided and undoubtedly, we believe, harmful to 
many students. Such a perspective thus reinforces an 
educational idiom in which students are encouraged 
to attribute academic failure or attendance at lower- 
ranking schools to lack of ability, rather than to other 
possible explanations pertaining to behaviours under 
an individual’s control (Dweck & Leggett, 1998). To 
this point, we note that several current research and 



The Sophistry of University Rankings 


163 


counseling programs are thus encouraging students 
to adopt more helpful and enlightened orientations 
to learning. The attribution of failure to lack of abil¬ 
ity has been linked to many undesirable educational 
outcomes, such as decreased task persistence, as well 
as more apathetic and more passive or ‘helpless’ ori¬ 
entations to learning (e.g., Dweck & Leggett, 1998; 
Dweck & Molden, 2005). 

Portrayals of smartness as a static and directly 
quantifiable ability are marketed within a culture 
modeled upon the metaphor of Consumer Reports — 
winners, losers, a ‘best of all’ school, and so on. More¬ 
over, the rapidly expanding interest and literature on 
‘self-handicapping’ (e.g., Urdan & Midgley, 2001), 
raises the researchable hypothesis for the future that 
students from lower-ranking schools and universities 
may engage in relatively higher levels of self-defeating 
and distracting activities, which in turn may serve ef¬ 
fectively as excuses for lower academic performance. 
Future research must therefore be directed to ques¬ 
tion whether students from lower-ranking institu¬ 
tions suffer in terms of job-seeking activities and even 
whether, for example, their academic output is evalu¬ 
ated differently or less positively relative to the same 
input when shown by non-stigmatized students. 

It is perhaps ironic that while Canadian 
schools have now (as of March, 2009) largely with¬ 
drawn their active support in terms of providing 
evaluative data upon demand to Macleans maga¬ 
zine, several have nevertheless drawn upon their rank 
standings, strategically and selectively, with which 
to promote and market particular programs within 
the business of higher education. In view of the per¬ 
ceived inevitability of ranking exercises, and the re¬ 
sulting prospect of publicity, several schools naturally 
have engaged in lobbying efforts toward inclusion of 
additional ranking indices perceived as likely to en¬ 
hance particularly their own relative rank standings. 

We do not expect customers to evaluate (or 
reliably rank) all cars in a used car lot, to choose or 
agree upon the criteria to be used for such a task, or 
to possess the knowledge required to attempt mean¬ 
ingful comparisons. We similarly would believe that 
incoming students, for the most part, are hardly in a 
credible position with which to evaluate and mean¬ 
ingfully calibrate indices of the type which underlie 


university ranking exercises. Moreover, such vulner¬ 
ability is increased even more in the midst of unre¬ 
lenting political and lobbying activities on the part 
of university authorities, as well by as the siren call of 
advertising, employed deliberately and competitively 
by many schools, with which to attract student-cus¬ 
tomers. The student, with an individual and unique 
worm’s eye view, is thus effectively a target, caught 
in the crossfire between the traditional ideals and 
presumed value of higher education, and the mun¬ 
dane reality of promotive marketing. In our opinion, 
and looking to the future, the characteristics of the 
underlying data, and the unintended but real conse¬ 
quences of annual ranking exercises, should now be 
of concern to parents, students, and authorities at all 
educational levels. 
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